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EVALUATING SPP/APR IMPROVEMENT ACTIVITIES 



Purpose and Target Audiences 

This document is intended to assist State Education Agency (SEA] and Lead Agency (LA] staff and 
technical assistance providers in designing a meaningful evaluation for the State Performance Plan 
(SPP] /Annual Performance Report (APR] improvement activities. It provides 1] information about 
the relevance of evaluation in the context of improvement planning and strategic systems thinking; 
2] guidance on selecting an appropriate design for evaluating different types of improvement 
activities and; 3] additional resources and tools that support the overall design, implementation 
and evaluation of the SPP which may serve as a State's blueprint for systems improvement. 

Systems Change and the SPP/APR 

"A system is a group of interacting, interrelated, and interdependent components that form a 
complex and unified whole." (Coffman, 2007, p. 3], A system's overall purpose or goal is achieved 
through the actions and interactions of its components. The Part C and Part B Programs are 
complex systems with interrelated and interdependent components. These systems have multiple 
policies, agencies, and strategies at the national, state and local level with varying outcomes at each 
of the different levels but with the common goal of achieving improved outcomes for children with 
disabilities and their families. 

Given that systems are interrelated and interconnected, "systems change involves changing the 
capacity, interrelationships and interdependencies among parts, levels and stakeholders of a 
system with the consequence that desired changes in one part and level are accompanied by 
changes in other parts and levels to reach an idealized and sustainable vision of the whole.” (Olsen, 
quoted in Hurth & Goode, 2009, p. 1] To illustrate these complex systems, one might use a 'theory 
of change' model to show the process by which change is expected to occur and the responsibility 
that a particular initiative plays in producing that change. Sometimes illustrated through a visual 
depiction or logic model, a 'theory of change' shows how the inputs (resources] and processes 
(activities] connect to outputs (products or units produced, such as number of staff trained] and 
outcomes (intended results]. Complex programs typically describe the intermediate outcomes 
(systems changes, including improved policies, procedures, and practices] and subsequent longer- 
term impacts (the goals of the program, such as improved outcomes for children]. While the 
system is focused on the ultimate goal — better impacts for the system's beneficiaries — it is just as 
important to identify intermediate outcomes that set the stage for longer-term impact. These 
intermediate outcomes relate to the system and are important measures of progress along the way. 
(Kahn, et al, 2009] 



Inputs 


— ► 


Process 


— ► 


Outputs 


— ► 


Outcomes 



It is important to note that a 'theory of change' model enables stakeholders to understand how 
initiatives are doing in relation to their intended outcomes and impacts; and thus, can be used to 
inform the development of strategies or activities, as well as designing the evaluation. Sustainable 
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systems change requires developing "a comprehensive, long-term plan for implementing change 
and strengthening the infrastructure needed to sustain change at all levels of the system 
(supportive leadership, policies and procedures, funding mechanisms, accountability measures]." 
(Hurth & Goode, 2009, p. 2] 

One might view a State Performance Plan as a long-term plan for systems change. Because the 
system components support the existing or current state and are interrelated, a change in one 
component is not likely to be sustained unless accompanied by supportive changes in all related 
components. Therefore, a combination of strategies addressing multiple levels of the system is 
more likely to lead to sustainable change and the desired outcomes. Given this, a State's 
improvement activities must be viewed as interconnected, supporting changes to the infrastructure 
that work together to achieve the desired outcomes rather than functioning as separate activities. 
For example, the Part C SPP/APR indicator 7 (45-day timeline] investigative questions illustrate the 
interconnectedness of the systems that ensure that IFSP meetings are conducted within 45 days — 
what systems supports are in place and what additional infrastructure is needed to support the 
change? 

• Do we have clear policies and procedures in place that support quality 
provider practices and efficiency in meeting the 45-day timeline? Do we have 
policies and procedures that interfere with meeting the 45-day timeline? 

• Do we have efficient and effective procedures for serving children in foster 
care, protective services and homeless children - including the timely 
sharing of information, obtaining consents and scheduling? 

• Are our agreements / contracts with other agencies and providers effective in 
ensuring the 45-day timeline is met? 

• Do we provide opportunities for providers to receive professional 
development on this requirement? 

• Is our monitoring and supervision adequate for this requirement (e.g., are we 
efficiently tracking caseloads and timelines in order to manage assignment of 
service coordinators/evaluation teams and scheduling and filling 
cancellations to ensure that we meet the 45-day timeline]? 

• Do we have adequate numbers of providers to conduct evaluations and 
service coordinators to coordinate the IFSP process? If not, what are we 
doing to recruit and retain qualified personnel? 

Adapted from: http://spp-apr-calendar.rrfcnetwork.org/explorer/view/id/345 

Through these types of questions, a state can identify and implement a combination of 
improvement activities that collectively will strengthen the infrastructure and lead to sustainable 
change. 
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Types of SPP/APR Improvement Activities 

Suggested types of SPP improvement activities are offered for consideration to guide the SEA/LA in 
designing priority activities that will ultimately yield improved student achievement and child and 
family outcomes. Data based decisions should drive the selection process and activities targeted 
should represent improvements the state believes will have a significant impact on improving 
indicator data. An SEA/LA might choose one or more activities that fall within a specific category. 
When determining the organizational structures, protocols and technical assistance and training 
necessary to support the improvement planning process, actions may be designed within the 
following categories: 

a. Improve data collection and reporting 

b. Improve systems administration and monitoring 

c. Build systems and infrastructures of technical assistance and support 

d. Provide technical assistance/training/professional development 

e. Clarify/examine/develop policies and procedures 

f. Program development 

g. Collaboration/coordination 

h. Evaluation 

i. Increase/Adjust FTE 

Following the implementation of SPP improvement activities, it becomes necessary to evaluate the 
extent to which the activities were implemented, whether or not the appropriate audience was 
reached, and the extent to which the activities achieved what was intended. Ultimately, what was 
their impact on achieving or maintaining compliance or improving results relative to the 
indicator (s]? Results of these data will drive decisions regarding which improvement activities 
need to be maintained, revised, eliminated or added. 

Selecting and Reviewing a Set of Improvement Activities 

States have developed a 6-year SPP and annually revisit those plans and revise as needed. This 
allows for an opportunity to step back to look at the existing set of improvement activities and 
judge the extent to which a specific combination of activities can reasonably be expected to result in 
the desired systems change or improvements to the indicator. The following guiding questions are 
intended to help a state select or review their existing set of improvement activities and determine 
if they are appropriate, comprehensive, and specific enough to address all relevant system 
components to improve the indicator and ensure sustainability. The focus of the questions is to 
encourage thinking of the improvement activities from a systems thinking approach. The following 
section outlines the specific steps in designing the evaluation. 

1. Was a root cause analysis conducted to drive the selection of the improvement activities? 
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2. Was a logical linkage established between the root causefs], data collection, analysis and 
proposed improvement activity outcome(s]? 

3. Do the activities reflect evidence-based practices (e.g., from other states, from TA&D 
Centers]? 

4. Does the improvement activity address more than one indicator? Is it linked to other 
SEA/LA improvement initiatives/priorities? 

5. Have partners been identified to assist in improvement activity implementation? 

6. Have specific action steps been developed to complete the improvement activity (e.g., task, 
activity, person(s] responsible, resources needed, timeline, etc.]? 

7. Have personnel been identified who will develop, implement, monitor and evaluate the 
improvement activity? 

8. Are the short-term and long-term outcomes identified for the improvement activity? 

9. Have methods to collect the outcome data been identified? 

10. Have data sources been identified? 

11. Has a data collection timeline been developed? 

12. How will the outcome data be reported? To whom? 

Steps for Evaluating Individual Improvement Activities 

Several key aspects of evaluation should be considered when planning the evaluation of state 
improvement activities including goals, methods, timelines, baseline data, planned data analysis, 
planned data use and individual responsibilities. Consider the following seven steps to help guide 
your evaluation planning: 

1. What is the goal or purpose? Or what questions] will your evaluation be trying to answer? 
Is it process oriented? Or outcome/impact oriented? 

2. What methods will you use for data collection? What are the primary data sources? How 
should data be collected? Is sampling required? If so, how will you sample? How will data 
quality be ensured? Choose an appropriate methodfs]: focus groups, observation, surveys, 
interview, existing data, etc. 

3. What are your timelines for collecting, analyzing and reporting the data? When would the 
results of the evaluation be most useful? 

4. What is your baseline data that you will monitor for improvement? 

5. How will the data be analyzed? What analytical techniques are available and most 
appropriate? 

6. How will the evaluation results be used? How should the findings be packaged and shared 
in order to be most useful? 

7. Who will be responsible for the various aspects of the evaluation? 
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Scenario 



Imagine your state is missing a considerable amount of data for the child outcomes C3/B7 
indicators. In your analysis of the system, you discover that a number of local programs are unclear 
about who is responsible for collecting outcomes data, when the data should be collected, when the 
data should be reported, which children should be included in the data, etc. Upon further 
exploration, you conclude that some of the written policies and procedures are unclear and not all 
local administrators and staff have been trained on the process. One specific procedure that is 
unclear relates to the roles and responsibilities of Part C staff and Section 619 staff when a child 
turns age 3 years and is transitioning from Part C to Part B special education services. 

Your state determines that three key improvement activities should collectively address these 
issues — 1] collaborate across State Part C and Section 619 agencies to clarify roles and 
responsibilities of local programs at the data collection point when the child exits Part C and enters 
Part B, 2] revise written policies and procedures about child outcomes data collection in order to 
add new details and to clarify the existing information (including roles and responsibilities of Part C 
and Section 619 school district staff), and 3) provide additional training on the child outcomes data 
collection and reporting process (to include the revised written policies and procedures). 

Improvement Activity 1: Collaboration 

Collaborate across State Part C and Section 619 agencies to clarify roles and responsibilities of local 
programs at the data collection point when the child exits Part C and enters Part B. 

1. What is the goal or purpose of the activity? What question(s) will the evaluation be trying to 
answer? Is it process oriented? Or outcome/impact oriented? 

Goal: To clarify roles and responsibilities of Part C and Section 619 school district 

staff in the collection and reporting of child outcomes data, specifically at 
the shared data point when a child turns age 3 and transitions into Part B 
services. 

Questions: To what extent did the collaboration result in clear descriptions of roles 

and responsibilities for Part C and Section 619 staff? 

Type of evaluation: This is an outcome evaluation 



2. What methods will be used for data collection? What are the primary data sources? How should 
data be collected? Is sampling required? If so, how will the State sample? 



Methods: Focused discussion: small group of local staff review the written roles and 

responsibilities for clarity. If needed, the group provides additional ideas 
for clarification. 

Data sources: Local staff feedback gathered during the discussion. 

Sample: 8-10 staff from local programs willing to review and provide feedback on 

the written roles and responsibilities. 



3. What are the timelines for collecting, analyzing and reporting the data? When would the results 
of the evaluation be most useful? 



Timelines: Prior to the dissemination of revised policies and procedures and the 

provision of training. 



Evaluating SPP/APR Improvement Activities - 12/16/2009 



Page 6 



4. What is your baseline data that you will monitor for improvement? 

Baseline: Local reports that roles and responsibilities are unclear. Absence of 

written roles and responsibilities for Part C and Section 619 staff. 

5. How will the data be analyzed? What analytical techniques are available and most appropriate? 

Analysis: Notes/transcript from the reviewer discussion will be analyzed to 

determine whether or not the roles and responsibilities for Part C and 
Section 619 are clear and, if not, how they could be improved. 

6. How will the evaluation results be used? How should the findings be packaged and shared in 
order to be most useful? 

Use: Feedback from reviewers will help ensure written roles and 

responsibilities are clear. The written roles and responsibilities will then 
be incorporated into the overall policies and procedures related to 
collecting and reporting outcomes data and will be incorporated into the 
training agenda/materials. 

7. Who will be responsible for the various aspects of the evaluation? 

Responsibilities: Part C Lead Agency and 619 State staff will identify a set of local 

providers/program staff to participate in the review, and subsequently 
discuss the reviewer feedback and implications for additional clarification 
of roles and responsibilities. 



Improvement Activity 2: Policies and Procedures 

Revise written policies and procedures about child outcomes data collection in order to add new 
details and to clarify the existing information. 

1. What is the goal or purpose of the activity? What questions] will the evaluation be trying to 
answer? Is it process oriented? Is it outcome/impact oriented? 



Goal: 



Questions: 



Type of evaluation: 



To improve the written policies and procedures so that local program staff 
find them clear and comprehensive regarding collecting and reporting child 
outcomes data. 

To what extent do local programs find the policies and procedures clear 
and comprehensive? Do staff understand their role in collecting and 
reporting child outcomes data? Do staff understand when to collect the 
data? Do staff understand when to report the data? Do staff understand 
what children are eligible for participation in data collection? etc. 

This is an outcome evaluation 



2. What methods will you use for data collection? What are the primary data sources? How 
should data be collected? Is sampling required? If so, how will the State sample? 
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Methods: Ask local program staff to complete an online questionnaire after 

disseminating the revised policies and procedures to see if they find the 
revised policies and procedures to be clear and complete and also to 
measure their understanding of the policies for collecting and reporting 
child outcomes data. 

Data sources: Program survey. 

Sample: No sampling. Collect data from all programs. 

3. What are your timelines for collecting, analyzing and reporting the data? When would the 
results of the evaluation be most useful? 

Timelines: Gather program survey data in the month immediately following the 

dissemination of the revised policies. 

4. What is your baseline data that you will monitor for improvement? 

Baseline: Reports from local programs that policies and procedures are unclear. 

5. How will the data be analyzed? What analytical techniques are available and most appropriate? 

Analysis: Descriptive statistics (e.g., mean, median, mode, standard deviation, range] 

can be used to show programs' feedback on whether the revised policies 
and procedures are clear and complete. Descriptive statistics can also be 
used to show programs staffs' understanding of the data collection and 
reporting policies and procedures. 

6. How will the evaluation results be used? How should the findings be packaged and shared in 
order to be most useful? 

Use: The questionnaire data should be used to help determine whether the 

policies and procedures are clear and complete and whether program staff 
are understanding the data collection and reporting policies. If the data 
show that programs still do not find the policies clear, they may need 
additional revisions. 

7. Who will be responsible for the various aspects of the evaluation? 

Responsibilities: State Part C Lead Agency and Section 619 staff will be responsible for the 

development and dissemination of the survey. State officefs] will process 
the data and share results back with the stakeholder group that revised the 
survey. Stakeholder group will discuss the results and determine if 
additional revisions need to be made to the policies. 

Improvement Activity 3: Training and Technical Assistance 

Provide training on the child outcomes data collection and reporting process (including the revised 

policies and procedures], 

1. What is the goal or purpose of the activity? What questionfs] will the evaluation be trying to 
answer? Is it process oriented? Is it outcome/impact oriented? 
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Goal: To increase local provider/program understanding of the child outcomes 

data collection and reporting process and specifically their roles and 
responsibilities in related to collecting and reporting the data. 

Questions: To what extent did the training result in staff being knowledgeable about 

their roles and responsibilities? 

Type of evaluation: This is an outcome evaluation 

2. What methods will the State use for data collection? What are the primary data sources? How 
should data be collected? Is sampling required? If so, how will the State sample? 

Methods: Ask training participants to complete a questionnaire at the end of the 

training in order to measure their knowledge of the child outcomes data 
collection and reporting process, and specifically, their roles and 
responsibilities related to collecting and reporting the data. 

Data sources: Participant survey. 

Sample: No sampling. Collect data from all training participants. 

3. What are the State's timelines for collecting, analyzing and reporting the data? When would the 
results of the evaluation be most useful? 

Timelines: Gather participant survey data immediately following the training. 

4. What is your baseline data that you will monitor for improvement? 

Baseline: Data about programs/staff not yet trained. Reports from local programs 

about not understanding the child outcomes data collection process. 

5. How will the data be analyzed? What analytical techniques are available and most appropriate? 

Analysis: Descriptive statistics can be used to show participants' knowledge of child 

outcomes data collection and reporting process. 

6. How will the evaluation results be used? How should the findings be packaged and shared in 
order to be most useful? 

Use: The training questionnaire data should be used to help determine how 

much the participants learned/understand the child outcomes data 
collection and reporting process. If the initial training results are poor, 
additional supports will need to be provided, additional trainings may be 
needed and/or the training agenda, materials, etc. may need to be revised. 

7. Who will be responsible for the various aspects of the evaluation? 

Responsibilities: State staff and trainers will be responsible for the development of the 

training survey. Trainers will disseminate the questionnaire, analyze the 
data and share results back with the State staff. State staff and trainers will 
discuss the results and determine if additional follow up with the program 
is needed. 
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Across All Improvement Activities - Long term outcomes 

1. What is the long term goal of all the improvement activities (collectively]? What questions] 
will the evaluation try to answer? Is the evaluation process oriented? Is the evaluation 
outcome/impact oriented? 

Long term goal: To increase statewide reporting of child outcomes data (and decrease the 

missing data in the child outcomes indicator] 

Questions: To what extent did the statewide reporting of child outcome data increase 

(the missing data decrease]? 

Type of evaluation: This is an outcome evaluation 

2. What methods will the State use for data collection? What are the primary data sources? How 
should data be collected? Is sampling required? If so, how will you sample? 

Methods: Calculate the percentage of missing data for the Annual Performance 

Report for C3/B7 to see if the amount of missing data has decreased since 
the previous year. 

Data sources: Data system/monitoring data. 

Sample: No sampling. Collect data from all programs. 

3. What are the timelines for collecting, analyzing and reporting the data? When would the results 
of the evaluation be most useful? 

Timelines: Programs will collect and report child outcomes data for the months 

following the dissemination of the revised policies and procedures. 

At the end of the year, the State will calculate the missing data for the 
Annual Performance Report and compare it to the previous year's 
percentage of missing data. 

4. What is your baseline data that you will monitor for improvement? 

Baseline: % of missing data for C3/B7 for the previous year. 

5. How will the data be analyzed? What analytical techniques are available and most appropriate? 

Analysis: A comparison of the overall percentage of missing data before and after the 

revised policies and procedures and trainings can be used to demonstrate 
improvement in staff implementation of the child outcomes data collection 
and reporting. Further analysis by programs that participated in the 
training is an additional level of analysis. 

6. How will the evaluation results be used? How should the findings be packaged and shared in 
order to be most useful? 

Use: Results will be reported in the next APR. Additionally, the overall 

improvement on missing data will be shared with the local programs to 
discuss improvements and discuss potential for more improvement. Local 
programs that continue to have missing data will receive additional TA. 
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7. Who will be responsible for the various aspects of the evaluation? 



Responsibilities: State staff will calculate the percentage of missing data for the months 

following the dissemination of revised policies and procedures and 
compare the statewide data with the previous year's data. State staff will 
share the data with the local programs and discuss improvement and 
potential for additional improvement. 



Conclusion 

The above guidelines and examples provide the reader with an overview of systems change 
thinking as related to the selection and evaluation of SPP/APR improvement activities. The final 
section of this document lists additional resources and guidance documents that provide greater 
depth into the topic. 
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Additional Resources/Tools 

SPP/APR Calendar Investigative Questions. States are encouraged to answer the investigative 
questions for each SPP/APR Indicator to assist with determining their technical assistance 
needs. Use this link to select an indicator to review its investigative questions: http://spp-apr- 
calendar.rrfcnetwork.org/techassistance.html 

State Systems Improvement Self Assessment and Planning Guide. [ http://spp-apr- 

calendar.rrfcnetwork.org/getfile/view/id/562 ] Mid-South Regional Resource Center developed 
this 15-page document, designed to be used as an assessment of State systems by State Part B 
and Part C staff and their stakeholders. It provides a detailed process for State Education 
Agencies (SEA] and Lead Agencies [LA] to follow that will guide improvement efforts relative to 
the APR indicators and other statewide initiatives. Its purpose is to assist States in 1] 
implementing a systematic approach to designing and refining improvement activities; 2] 
aligning improvement activities with State priorities; 3] allocating funding and human 
resources to State improvement efforts which are most effective; and 4] implementing, 
tracking, documenting and evaluating improvement efforts designed to achieve or maintain 
compliance and improve performance that lead to better results. 

A Systems Framework to Guide Improvement Planning and Technical Assistance to States: 

Considering Multilevel Influences for Successful and Sustainable Implementation of 

Improvement Activities, [http://spp-apr-calendar.rrfcnetwork.org/getfile/view/id/688] 

The purpose of this guide is to assist states and TA providers in making improvement planning 
and technical assistance decisions within a systems context as it applies to the SPP/APR 
development. 

NCRRC SPP/APR Improvement Activity Review Form. [ http://spp-apr- 

calendar.rrfcnetwork.org/getfile/view/id/647 ] This form is intended for states to broadly 
assess quality of improvement activities for SPP/APR indicators. 

NECTAC TA Model [ http://spp-apr-calendar.rrfcnetwork.org/getfile/view/id/642 ] This 18-page 
document describes components of a state's Systems Change Plan. The content is presented in 
bulleted format for ease of use by TA providers working with an individual state on a process 
for improving the state system of services. 

How to Develop a SPDG Evaluation Plan. [P. Gonzalez, 2009] 

[http://www.signetwork.org/Docs/CallNotes/2009/FY09SPDGNoticeBiddersWebinar/PG_Eval 
uationPlanApr09.ppt] Although this PowerPoint focuses on State Personnel Development 
Grants, it does provide concrete steps in developing an evaluation plan. 

University of Wisconsin - Extension Program Development and Evaluation Sample Logic 
Models, [http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html ] 

W.K. Kellogg Foundation Logic Model Development Guide. 

[http://www.wkkf.org/Default.aspx?tabid=1185&NID=140&AddToCart=2813669;0&Language 
=0&ReturnURL=%2fdefault.aspx%3ftabid%3dl01%26CID%3d281%26CatID%3d281%26Item 
ID%3d2813669%26NID%3d20%26LanguageID%3d0&LanguageID=0] This guide was 
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developed to provide practical assistance to nonprofits engaged in the process of outcomes- 
based evaluation in orienting them to the underlying principles of "logic modeling". 
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